ERR non-ISO formats don't show position of error #50366

MarcoGorelli · 2022-12-20T21:17:43Z

closes ERR non-ISO formats don't show position of error #50361 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

MarcoGorelli · 2022-12-20T22:07:11Z

pandas/_libs/tslibs/strptime.pyx

+        except (ValueError, OutOfBoundsDatetime) as ex:
+            ex.args = (str(ex) + f" at position {i}", )


similar to

pandas/pandas/_libs/tslib.pyx

Lines 789 to 790 in 2f54a47

except (ValueError, OverflowError) as ex:

ex.args = (f"{ex} present at position {i}", )

I just removed the word 'present' as I don't think sounds good with the errors from this file

i.e.

time data ' ' does not match format '%m/%d/%Y' \(match\) at position 2

sounds better than

time data ' ' does not match format '%m/%d/%Y' \(match\) present at position 2

Nit but why not use a full f-string instead of addition between 2 strings here?

WillAyd

So IIUC the message is a bit different between ISO / non-ISO (the former has the word present?). Can we align the messages?

WillAyd · 2022-12-20T22:38:09Z

pandas/_libs/tslibs/strptime.pyx

+        except (ValueError, OutOfBoundsDatetime) as ex:
+            ex.args = (str(ex) + f" at position {i}", )


Nit but why not use a full f-string instead of addition between 2 strings here?

MarcoGorelli · 2022-12-21T10:37:31Z

pandas/tests/tools/test_to_datetime.py

        )
        with pytest.raises(OutOfBoundsDatetime, match=msg):
            with tm.assert_produces_warning(warning, match="Could not infer format"):
-                to_datetime("2417-10-27 00:00:00", format=format)
+                to_datetime("2417-10-10 00:00:00", format=format)


slightly changing the input here so we can parametrise of it with both "%Y-%m-%d %H:%M:%S" and "%Y-%d-%m %H:%M:%S" (to check ISO vs non-ISO)

MarcoGorelli · 2022-12-21T10:38:53Z

I've aligned them now, thanks - this should make the diff in #50242 much smaller

In [1]: to_datetime(['2020-01-01', '9999-01-01'], format='%Y-%m-%d')
---------------------------------------------------------------------------
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 9999-01-01 00:00:00 present at position 1

In [2]: to_datetime(['2020-01-01', '9999-01-01'], format='%Y-%d-%m')
---------------------------------------------------------------------------
OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 9999-01-01 00:00:00 present at position 1

In [3]: to_datetime(['2020-01-01', 'foo'], format='%Y-%m-%d')
---------------------------------------------------------------------------
ValueError: time data "foo" at position 1 doesn't match format "%Y-%m-%d" (match)

In [4]: to_datetime(['2020-01-01', 'foo'], format='%Y-%d-%m')
----------------------------------------------------------------------------
ValueError: time data "foo" at position 1 doesn't match format "%Y-%d-%m" (match)

jbrockmendel · 2022-12-21T23:19:48Z

pandas/tests/tools/test_to_datetime.py

@@ -1133,10 +1139,10 @@ def test_datetime_invalid_scalar(self, value, format, warning):
        assert res is NaT

        msg = (


nitpick can you use msg = "|".join(...) instead of a single string

jbrockmendel · 2022-12-21T23:20:27Z

pandas/_libs/tslib.pyx

                            raise ValueError(
                                f"time data \"{val}\" at position {i} doesn't "
-                                f"match format \"{format}\""
+                                f"match format \"{format}\" ({match_msg})"


ending this with "search" seems weird (and i dont see it in any test cases)

It's true that it wasn't tested, but it's been like this since pandas 0.24:

In [1]: import pandas In [2]: pandas.__version__ Out[2]: '0.24.0' In [3]: pandas.to_datetime(['2020-01-f00'], format='%Y-%d-%m', exact=False) --------------------------------------------------------------------------- ValueError: time data '2020-01-f00' does not match format '%Y-%d-%m' (search)

Maybe we can just remove it - people typically know whether they've passed exact, and it doesn't really add much

Especially as it was never there for (more common) ISO formats:

In [4]: pandas.to_datetime(['2020-01-f00'], format='%Y-%m-%d', exact=False) --------------------------------------------------------------------------- ValueError: time data 2020-01-f00 doesn't match format specified

Let's just unify them to

time data '2020-01-f00' does not match format '%Y-%d-%m'

…r-message

WillAyd

looks pretty good

WillAyd · 2022-12-24T19:36:10Z

pandas/_libs/tslibs/strptime.pyx

@@ -396,7 +401,15 @@ def array_strptime(

            result_timezone[i] = tz

-        except (ValueError, OutOfBoundsDatetime):
+        except OutOfBoundsDatetime as ex:


Rather than breaking off this branch can we not keep the existing one and just specialize for the OOB exception within?

you're right, that's better - thanks!

mroeschke

LGTM

WillAyd

awesome thanks @MarcoGorelli

WillAyd · 2022-12-27T21:44:33Z

Looks like a merge conflict but feel free to merge after that

MarcoGorelli · 2022-12-27T21:45:19Z

cool, thanks for reviewing!

jbrockmendel · 2022-12-27T22:13:51Z

pandas/tests/tools/test_to_datetime.py

@@ -1882,7 +1891,10 @@ def test_dataframe_mixed(self, cache):
    def test_dataframe_float(self, cache):
        # float
        df = DataFrame({"year": [2000, 2001], "month": [1.5, 1], "day": [1, 1]})
-        msg = "cannot assemble the datetimes: unconverted data remains: 1"
+        msg = (
+            r"^cannot assemble the datetimes: unconverted data remains at position "


NBD but is the "r" necessary?

Shouldn't be, it's just a habit to always use that in regular expressions

wip

303a648

MarcoGorelli changed the title ~~Note position in err message~~ ERR non-ISO formats don't show position of error Dec 20, 2022

MarcoGorelli marked this pull request as draft December 20, 2022 21:18

MarcoGorelli force-pushed the note-position-in-err-message branch from 464aa84 to a2cff0b Compare December 20, 2022 21:35

MarcoGorelli marked this pull request as ready for review December 20, 2022 21:35

MarcoGorelli force-pushed the note-position-in-err-message branch from a2cff0b to 303a648 Compare December 20, 2022 21:58

MarcoGorelli added Error Reporting Incorrect or improved errors from pandas Datetime Datetime data dtype labels Dec 20, 2022

MarcoGorelli commented Dec 20, 2022

View reviewed changes

MarcoGorelli requested review from WillAyd, mroeschke and jbrockmendel December 20, 2022 22:07

MarcoGorelli mentioned this pull request Dec 20, 2022

BUG refactor datetime parsing and fix 8 bugs #50242

Merged

12 tasks

WillAyd reviewed Dec 20, 2022

View reviewed changes

MarcoGorelli marked this pull request as draft December 20, 2022 23:18

MarcoGorelli added 2 commits December 21, 2022 10:27

unify messages

8e40df4

fixup regexes

9040dca

MarcoGorelli marked this pull request as ready for review December 21, 2022 10:34

simplify

c49640d

MarcoGorelli commented Dec 21, 2022

View reviewed changes

MarcoGorelli requested a review from WillAyd December 21, 2022 10:42

fix test

1e7613f

jbrockmendel reviewed Dec 21, 2022

View reviewed changes

MarcoGorelli and others added 4 commits December 22, 2022 09:09

Merge remote-tracking branch 'upstream/main' into note-position-in-er…

93c3026

…r-message

remove (search) and (match) from messages

3338b43

Merge branch 'main' into note-position-in-err-message

ba48f33

simplify

34a7138

WillAyd reviewed Dec 24, 2022

View reviewed changes

mroeschke approved these changes Dec 27, 2022

View reviewed changes

MarcoGorelli requested review from WillAyd and jbrockmendel December 27, 2022 19:16

WillAyd approved these changes Dec 27, 2022

View reviewed changes

Merge branch 'main' into note-position-in-err-message

8543fc9

jbrockmendel reviewed Dec 27, 2022

View reviewed changes

MarcoGorelli added this to the 2.0 milestone Dec 28, 2022

MarcoGorelli merged commit 4e88c3f into pandas-dev:main Dec 28, 2022

MarcoGorelli mentioned this pull request Dec 28, 2022

ERR: "day out of range" doesn't show position of error #50464

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ERR non-ISO formats don't show position of error #50366

ERR non-ISO formats don't show position of error #50366

MarcoGorelli commented Dec 20, 2022 •

edited

Loading

MarcoGorelli Dec 20, 2022

WillAyd Dec 20, 2022

WillAyd left a comment

WillAyd Dec 20, 2022

MarcoGorelli Dec 21, 2022

MarcoGorelli commented Dec 21, 2022 •

edited

Loading

jbrockmendel Dec 21, 2022

jbrockmendel Dec 21, 2022

MarcoGorelli Dec 22, 2022 •

edited

Loading

WillAyd left a comment

WillAyd Dec 24, 2022

MarcoGorelli Dec 24, 2022

mroeschke left a comment

WillAyd left a comment

WillAyd commented Dec 27, 2022

MarcoGorelli commented Dec 27, 2022

jbrockmendel Dec 27, 2022

MarcoGorelli Dec 27, 2022

		except (ValueError, OutOfBoundsDatetime) as ex:
		ex.args = (str(ex) + f" at position {i}", )

	except (ValueError, OverflowError) as ex:
	ex.args = (f"{ex} present at position {i}", )

		@@ -1133,10 +1139,10 @@ def test_datetime_invalid_scalar(self, value, format, warning):
		assert res is NaT

		msg = (

ERR non-ISO formats don't show position of error #50366

ERR non-ISO formats don't show position of error #50366

Conversation

MarcoGorelli commented Dec 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli commented Dec 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli Dec 22, 2022 • edited Loading

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd commented Dec 27, 2022

MarcoGorelli commented Dec 27, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MarcoGorelli commented Dec 20, 2022 •

edited

Loading

MarcoGorelli commented Dec 21, 2022 •

edited

Loading

MarcoGorelli Dec 22, 2022 •

edited

Loading